Automated calibration of smartphone cameras for 3D reconstruction of mechanical pipes

This paper outlines a new framework for the calibration of optical instruments, in particular smartphone cameras, using highly redundant circular black-and-white target fields. New methods were introduced (i) matching targets between images; (ii) adjusting systematic eccentricity error centres; and (iii) iteratively improving solution through free-network self-calibrating bundle adjustment. The proposed method effectively matched 270 images, taken within laboratory, with robustness to type II errors (false negatives). adjustment, which requires only camera projective matrices from two views, behaved comparably available closed-form solutions, require additional priori object-space information. Finally, specifically case mobile devices, parameters obtained found be superior compared situ estimating 3D reconstructed radius mechanical pipe (approximately 45% improvement on average). Cet article présente une nouvelle procédure pour l’étalonnage d’instruments optiques, en particulier les caméras de smartphones, utilisant des champs hautement redondants cibles circulaires noir et blanc. De nouvelles méthodes sont introduites apparier entre corriger l’erreur d’excentricité systématique sur centres cibles; améliorer manière itérative la d’étalonnage grâce à un ajustement faisceaux d’auto-étalonnage réseau libre. Cette apparié efficacement dans images prises avec smartphones laboratoire d’étalonnage, bonne robustesse aux erreurs (faux négatifs). La méthode proposée correction d’excentricité, qui nécessite seulement projectives deux vue, s’est comportée comparable solutions forme fermée disponibles, nécessitent informations supplémentaires cible l’espace objet. Enfin, le cas appareils mobiles, paramètres obtenus l’aide cette se avérés meilleure qualité que par estimer rayon d'un tuyau mécanique reconstruit (amélioration d’environ moyenne). Dieses Manuskript bietet einen neuen Rahmen für die Kalibrierung optischer Instrumente, insbesondere von Smartphone-Kameras, unter Verwendung hochredundanter kreisförmiger Schwarz-Weiß-Zielfelder. Es wurden neue Methoden eingeführt den Abgleich Zielen zwischen Bildern, Anpassung systematischen Exzentrizitätsfehlers der Zielzentren und iterative Verbesserung Kalibrierungslösung durch eine selbstkalibrierende Bündel Anpassung. wurde beobachtet, dass vorgeschlagene Zielanpassung effektiv kreisförmige Ziele Smartphone-Bildern aus einem Kalibrierungslabor mit Robustheit gegenüber Typ-II-Fehlern abgleicht. Die Exzentrizitätsanpassung, nur projektive Kameramatrizen zwei Ansichten benötigt, verhielt sich synonym zu verfügbaren Lösungen geschlossener Form, mehrere zusätzliche Objektraum-Zielinformationen erfordern. Schließlich, speziell Fall Smartphone-Geräten, Kalibrierungsparameter unserem erhalten gefunden überlegen im Vergleich in-situ-Kalibrierung Schätzung 3D-rekonstruierten Radius eines mechanischen Rohres (ca. Durchschnitt). Este manuscrito proporciona marco conceptual nuevo para calibración instrumentos ópticos, cámaras teléfonos inteligentes, utilizando dianas circulares blanco y negro altamente redundantes. Se introdujeron nuevos métodos correspondencia imágenes; el ajuste del excentricidad sistemático los centros las dianas; mejora iterativa solución mediante libre con autocalibración. El método propuesto realizó efectiva imágenes teléfono inteligente, tomadas laboratorio calibración, robustez errores tipo (falsos negativos). propuesto, solo requiere proyectivas cámara desde dos orientaciones, comportó manera soluciones forma cerrada requieren información adicional espacio objeto. Finalmente, específicamente caso dispositivos móviles, parámetros obtenidos este conceptual, comparación situ, proporcionaron mejor estimación radio una tubería mecánica reconstrucción (aproximadamente promedio). 本文为光学仪器（尤其是智能手机相机）的检校提供了一种新方法，该方法使用了高度冗余的圆形黑白检校目标，引入了用于（i）在图像之间匹配目标的新方法；（ii）调整目标中心的系统性偏心误差；（iii）通过自由网自检校光束法平差来迭代地改进检校方案。所提出的方法有效地匹配了在检校实验室中拍摄的270个智能手机图像中的圆形目标，并具有对II型误差（假阴性）的鲁棒性。所提出的目标偏心调整仅需要来自两幅影像的相机投影矩阵，其性能与现有的封闭式解决方案相类似，后者需要附加的目标在物方空间的信息。最后，利用移动设备(相机)来计算估计机械管道三维重建的半径，使用本方法获得的校准参数要优于现场校准的结果(平均提高约45％)。 To find exact point correspondences, targets, such as those shown laboratory Fig. 1, are almost exclusively utilised high-precision close-range photogrammetry applications (Luhmann, 2014). Circular geometrically approximated by ellipses images. They offer several advantages unique centre, invariance under rotation translation, low cost production. As number well increases, one hand manual corresponding becomes more tedious, time consuming impractical. On other hand, automated geometric characteristics ellipses, runs risk mismatches (type errors, also termed false negatives or omission errors). Therefore, coded recommended reduce effects during (Shortis Seager, Coded however, still design distinct identifier labelling individually, may serve impractical large A fully process identify match minimal intervention is, hence, desirable larger fields In addition, even though object space projected image plane, best-fitting do not necessarily correspond actual centre target. is commonly referred error, consequence geometry. suggests that absence random will original Hence, must corrected, especially requiring metrology. important consideration selection appropriate model. Especially newly released extent impact parameters, including terms required correct radial lens distortions, evaluated. this study goal model IOPs requirements metric 4K video recordings, acquired three latest namely, iPhone 11, Huawei P30 Samsung S10. Video recordings since broader application pertains progress monitoring pipes construction projects, practical recording (as opposed single images). real-world cylinders representing pipes, support as-built documentation sites, examined. review previous literature has been divided into main categories: conics models cameras. These further explained following sections. Fundamental widely implemented computer vision provide an algebraic representation epipolar geometry Given sufficient set points (at least seven), fundamental matrix ( F) can estimated (Hartley Zisserman, 2004). An property F views invariant transformation space. relative directly up ambiguity (Luong Viéville, 1996). If assumed unchanged reconstruction possible affine ambiguity. initial estimate available, decomposed essential matrix, E = K T , where represents intrinsic (IOPs). K, orientation cameras retrieved singular-value decomposition 2004) five correspondences (Stewénius al., 2006). cases, it recover similarity (an arbitrary scale factor). two-view extended multiple views. fact, m recovered, given at - 1 pairwise epipoles, factorisation Sturm Triggs (1996). latter example global framework. practice, sequential registration typically produces reliable results, due flexibilities control inherent incremental (Schönberger, 2018). output SfM EOPs, feature points, sparse clouds. Several approaches, software packages, exist perform different variants SfM. study, COLMAP, open-source package comprised many computational scientific improvements traditional methods, documented Schönberger (2018), was utilised. Smartphone considered pinhole collinearity condition used straight-line relationship observed x, y), its homologous X, Y, Z) perspective X c Y Z ), described Luhmann al. (2014). Random departures hypothesised modelled additive, zero-mean noise ε x ) while Δ y) represent terms. comprise distortion decentring distortion. Radial distortion, far most often standard polynomial (such 2016), higher-order have demonstrated wide-angle lenses (Lichti 2020). Images collected modern generally corrected However, impact, if any, known subject investigation. Each section detail focuses sequences. improve precision self-calibration prevent compensation (coupling), must: capture depth variation field; convergent; rotate 90° about camera’s axis (thus both landscape portrait One corner complete (Fig. 1(a)), consists attached right-angled intersecting walls. planar walls create field. ladder collect convergent scene, starting bottom facing ceiling 1(b)) ending top floor 1(c)). At top, rotated then recorded reverse order (Figs. 1(d) (e)). problem requires: estimation 2(a)); detection each 2(b)); 2(c)); (iv) ellipses’ 2(d)). steps discussed below. reference, 2(a) illustrates positions orientations sequence strategy Algorithm 1. 3 shows refinement features observed, mismatched features, represented red green circles, correctly removed. circle indicates did satisfy step 2 whereas indicated removed 3. example, marginal, out around 300 matches negatively affect results when contain mismatches. robust ellipse presented Maalek Lichti (2021b) detect non-overlapping targets. I positives; commission errors) negatives; errors), established Fornaciari (2014) Pǎtrǎucean (2017). Once detected, parameter vector confocal hyperbola fitting method, (2021a). (centre, semi-major length, semi-minor length angle) converted form, transformed form conic equation (1). Equation (1) here verify brute-force checking correspondence condition, (1), every detected all become computationally expensive calculated equal zero presence measurement (Quan, threshold uncertainties errors. arbitrarily selected will, guarantee either no Step 4(b) performs inlier lie near line. Furthermore, instead choosing Δ, minimum used. (mismatching targets), triangulation LMedS performed. Modelling investigation, metrology (Ahn 1999; He 2013; Luhmann, Closed formulations plane (Dold, 1996; Ahn 1999). knowledge target’s target, coordinates normal plane. information, cannot trivially retrieved. Even external exists, constraint undesirable practices, focus study. following, retrieve projection true onto image, fed adjustment estimates EOPs determined, again adjusted recursively refine required/satisfactory precision. before (after 4) so incorrectly rejected (improving matching). 4 2. (amongst total 23 overlapping targets) applying final algorithm accepted methodology obtaining highest accuracy (Luhmann 2016). Provided aforementioned measures incorporated imaging network, successfully decorrelated and, thus, recovered accurately. Any outlier observations stage algorithm, least-squares estimation. (which network-invariant – per camera), estimated. singularity normal-equations caused datum defect adding inner constraints (free-network 2014)) yields optimal Following quality examined computed quantities. Most among these their estimates, together derived correlation coefficient quantify success decorrelation. residuals crucial graphically statistically assessing effectiveness modelling. unmodelled readily identified plot component function distance principal point. Moreover, quantified comparing photogrammetrically determined (or distances between) reference values independent source. calibration. system affected distortions significantly impacted choice included augmented condition. aim trade-off goodness-of-fit allowable bias. avoid underparameterised insufficient describe profile lead propagation bias optimistic overparameterisation than necessary introduce correlations variables inflate turn, degrade accuracy. end, similar (2020) employed. Using performed without any parameters. interior distance. Graphical analyses residuals, supported statistical testing information criteria, make informed decision coefficients added. particular, v r plotted point, r, assessed. added recomputed. repeated until trends remain. 1.5 minutes videos 30 frames second (fps) data collection above P30, 11 S10 smartphones. portion contained 130 instrument captured fill frame same consistency. Open Camera app utilised, possible, help access raw configurations, infinity, disabling autofocus fixing exposures. From footage, 90 fps decomposed: frameworks Algorithms calibrate convergence angles 80°, 88° 109° networks, respectively. IOPs, Table I. Four experiments designed assess namely: evaluation incorporating (impact comparison error); assessment matching; reconstruction. four following. developed transformation. experiment evaluate settings. 28 pre-calibrated P30. view COLMAP 5(a). ground-truth manually pronounced black cross 5(b)). 100 mm average 530 defined 0.13 mm. sample Two to: centre; performance closed formulation Here, N 50 combinations selected. provided (2014), view, plane’s normal. determine Luhmann’s formula, radius, normal, For (in pixels) authors’ (Algorithm 3) formula. reported. reported Resulting Calibration Parameters Conditions, extracted (2021b). settings: triangulation; centres. objective pipes. mock professionally installed 6. interest after pre-calibration modelling COLMAP’s default process. involves automatic calibration, first exchangeable file (Exif) IOP process, on, COLMAP-radial. 60-second 30-second fps), mode mode. height COLMAP-radial reasonable opportunity instruments coupling. dense carried once target-based cylinder scaled 6(c)) TLS cloud 6(d)) (2019). root mean square (RMSE) cylinder, uncertainty, 0.3 57.3 mm, TLS, approximately 0.1 consistent sub-millimetre (2019a). comparison. summary II. (shown 5), 3, increased. 7 (for combinations) blue) red). unadjusted centres, visually remains constant increases 5 28. better 0.64 9.46 average, 15 times result demonstrates reaching image. done formulation, absolute deviation (here “relative eccentricity”) 8 difference methods. illustrated, comparable, achieved slightly best 22 worst six 2.84 2.41 pixels but 20% method. attractive requirement formulations. precision, recall, F-measure (before 6); (with triangulation); combined adjust III. identified. considerably triangulation, mismatching (no relatively lower some reduced, contributing increase positives). When step) maintained (recall 100%), eliminated 3% precision). F-measure, provides value explain contributions combination result. use eliminate triangulation. videos, fps, sizes. utilises potential scene boundaries viewed another Self-calibrating re-estimate re-introduced iteration found). via showed accuracy, revealed that, long It (and cases outperformed) correcting opportunities utilise third assessed algorithm. (enhancing last evaluated pipework research (around 1.3 better). show advantageous. Specifically, achieve pipe-radius results. authors wish acknowledge MJS Mechanical Ltd Michael Baytalan professional installation purpose project partly funded Natural Sciences Engineering Research Council (NSERC) Canada (542980-19) Alberta Innovates (G2020000051). funding enabled organized ProjektDEAL.